docs: document gh workflow run for integration tests in ADDINGMODEL.md#2783
docs: document gh workflow run for integration tests in ADDINGMODEL.md#2783juanmichelini merged 2 commits intomainfrom
Conversation
Co-authored-by: openhands <openhands@all-hands.dev> Updated the workflow order to: 1. Create draft PR first 2. Run integration tests 3. Fix issues if needed 4. Mark PR ready when tests pass Also updated Step 7 to include the gh workflow run command with proper parameters (model_ids, issue_number, reason, ref).
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
all-hands-bot
left a comment
There was a problem hiding this comment.
🟡 Acceptable - Solves the real problem (unclear workflow → automated command), eliminates manual UI clicking, makes the process scriptable. Good pragmatic improvement.
| ## Step 6: Create Draft PR | ||
|
|
||
| **Mandatory**: Integration tests must pass before creating PR. | ||
| Push your branch and create a draft PR. Note the PR number returned - you'll need it for the integration tests. |
There was a problem hiding this comment.
🟡 Suggestion: You say "create a draft PR" but don't show how. Add the command:
gh pr create --draft --title "Add model: your-model-id" --body "Fixes #<issue-number>"Then users can actually capture the PR number from the output.
| ``` | ||
|
|
||
| ### Required in PR Description | ||
|
|
There was a problem hiding this comment.
🟡 Suggestion: Since workflow results are now auto-posted as PR comments (line 263), is the manual "Integration test results" link still required in the PR description? Either:
- Update this section to say the run URL is optional (workflow posts it automatically), or
- Clarify that contributors should still add it manually for visibility
Right now it's unclear whether this is redundant with the automated comment.
all-hands-bot
left a comment
There was a problem hiding this comment.
✅ QA Report: PASS WITH ISSUES
Summary: Documentation change successfully improves the integration test workflow by documenting the gh workflow run command. All documented commands are valid and parameters match the actual workflow.
Environment Setup
✅ Repository checked out successfully
✅ Workflow file .github/workflows/integration-runner.yml exists
✅ All referenced sections in the document exist
CI & Test Status
ℹ️ No CI checks configured for this documentation-only change (expected)
Functional Verification
1. Workflow Command Syntax ✅
Verified that the documented gh workflow run command syntax is correct:
$ gh workflow run --help
# Confirmed: -f/--raw-field for inputs, --ref for branch selection2. Workflow Parameters ✅
Verified all documented parameters match the workflow file:
model_ids- ✅ exists in workflow (required: false)reason- ✅ exists in workflow (required: true)issue_number- ✅ exists in workflow (required: false)
Workflow input definition:
workflow_dispatch:
inputs:
model_ids: { required: false, type: string }
reason: { required: true }
issue_number: { required: false, type: string }3. Comment Posting Feature ✅
Verified that the workflow does post results to PR when issue_number is provided:
post-dispatch-comment:
if: github.event_name == 'workflow_dispatch' && github.event.inputs.issue_number != ''
env:
ISSUE_NUMBER: ${{ github.event.inputs.issue_number }}4. gh pr ready Command ✅
Verified the documented command exists and syntax is correct:
$ gh pr ready --help
# Confirmed: Command exists and accepts PR number as argument5. Document Structure ✅
- Markdown syntax is valid
- All internal references exist (e.g.,
#common-issues) - Code blocks are properly formatted
- Step numbering is sequential (6→7→8→9)
Issues Found
1 Minor Issue - See inline comment on line 245
Verdict
Testing Evidence:
- Verified workflow file exists and has been triggered 4070+ times with
workflow_dispatchevent - Confirmed all
ghCLI commands are valid via help output - Validated all workflow parameters against actual workflow definition
- Checked internal document references resolve correctly
| ## Step 6: Create Draft PR | ||
|
|
||
| **Mandatory**: Integration tests must pass before creating PR. | ||
| Push your branch and create a draft PR. Note the PR number returned - you'll need it for the integration tests. |
There was a problem hiding this comment.
🟡 Minor: Step 6 instructs users to "Note the PR number returned" but doesn't show the command to create a draft PR.
Current state: Users might create the PR via web UI where the number isn't explicitly "returned".
Suggestion: Consider adding the actual command for clarity:
| Push your branch and create a draft PR. Note the PR number returned - you'll need it for the integration tests. | |
| Push your branch and create a draft PR: | |
| ```bash | |
| gh pr create --draft --title "Add your-model-id" --body "WIP: Adding new model" | |
| # Note the PR number returned - you'll need it for the integration tests |
This would make the workflow fully CLI-based and consistent with Steps 7 and 9. However, this is acceptable as-is since developers working on model additions likely know how to create PRs.
Problem
When adding a new model to
resolve_model_config.py, integration tests were mentioned but the process to trigger them was unclear and manual. This led to PRs being created without running the integration tests first.Solution
Updated
ADDINGMODEL.mdto document thegh workflow runcommand for triggering integration tests with the correct parameters:The workflow order was adjusted to:
This ensures integration tests are explicitly triggered and results are posted back to the PR.
Fixes #2785
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:3712bd7-pythonRun
All tags pushed for this build
About Multi-Architecture Support
3712bd7-python) is a multi-arch manifest supporting both amd64 and arm643712bd7-python-amd64) are also available if needed